Using Con dence Interval to Summarize the Evaluation Results: A Case Study

Distributed Shared Memory(DSM) has gained popular acceptance by combining the scalability and low cost of distributed system with the ease of use of single address space. Many new hardware DSM and software DSM systems were proposed in recent years. In general, benchmarking is widely used to demonstrate the performance advantages of new systems. However, the common method used to summarize the measured results is the arithmetic mean of ratios, which is incorrect in some cases. Furthermore, many published papers showed a lot of data only, and did not summarize them e ectively, which confused users greatly. In fact, many users want to get a single number as conclusion, which was not provided in old evaluation methods. Therefore, a new data summarizing technique based on con dence interval is proposed in this paper. The new technique includes two data summarizing methods: (1) paired con dence interval method; (2) unpaired con dence interval method. With this new technique, we can say at some con dence that one system is better than others. On the other hand, with the help of con dence level, we propose to standardize the benchmarks used for evaluating DSM systems so that we can get a convincible result. Moreover, our new summarizing technique ts not only for evaluating DSM systems, but also for evaluating other systems, such as memory system and communication systems.